The following applications are related to the present application, and are hereby incorporated by reference as though fully and completely set forth herein:
Ser. No. 08/340,667 titled xe2x80x9cIntegrated Video and Memory Controller With Data Processing and Graphical Processing Capabilitiesxe2x80x9d and filed Nov. 16, 1994 (5143-00100)
Ser. No. 08/463,106 titled xe2x80x9cMemory Controller Including Embedded Data Compression and Decompression Enginesxe2x80x9d and filed Jun. 5, 1995 (5143-00200)
Ser. No. 08/916,464 titled xe2x80x9cMemory Controller Including Embedded Data Compression and Decompression Enginesxe2x80x9d and filed Aug. 8, 1997 (5143-00201)
Ser. No. 08/522,129 titled xe2x80x9cMemory and Graphics Controller Which Performs Pointer-Based Display List Video Refresh Operationsxe2x80x9d and filed Aug. 31, 1995 (5143-00300)
Ser. No. 08/565,103 titled xe2x80x9cMemory and Graphics Controller Which Performs Pointer-Based Display List Video Refresh Operationsxe2x80x9d and filed Nov. 30, 1995 (5143-00301)
Ser. No. 08/770,017 titled xe2x80x9cSystem and Method for Simultaneously Displaying a Plurality of Video Data Objects Having Different Bit Per Pixel Formatsxe2x80x9d and filed Dec. 19, 1996 (5143-00302).
Ser. No. 08/604,670 titled xe2x80x9cGraphics System Including a Virtual Frame Buffer Which Stores Video/Pixel Data in a Plurality of Memory Areasxe2x80x9d and filed Feb. 21, 1996 (5143-00303).
Ser. No. 09/291,366 titled xe2x80x9cGraphics System and Method for Rendering Independent 2D and 3D Objects Using Pointer Based Display List Video Refresh Operationsxe2x80x9d and filed Apr. 14, 1999 (5143-01101).
Ser. No. 09/056,021 titled xe2x80x9cVideo/Graphics Controller Which Performs Pointer-Based Display List Video Refresh Operationsxe2x80x9d and filed Apr. 6, 1998 (5143-01200).
The present invention relates to system video/graphics system architectures, and more particularly to a video/graphics controller which performs pointer-based display list refresh operations to transfer video data from a memory to a display device, such as a television screen or a video monitor.
Digital display devices such as computer systems and digital televisions generally include a memory area, often referred to as a frame buffer, which stores the image or video portion which is currently being displayed. For example, in a computer system, the frame buffer is typically stored in a separate VRAM memory, or in the system memory. The graphics or video controller device reads the pixel data stored in the frame buffer and in turn generates the appropriate video signals to drive the display monitor. In a similar manner, digital television systems include a memory which serves as a frame buffer, wherein the memory stores the current image or video portion being displayed, or stores an inset or subset image which is being displayed in a larger image.
Computer systems, digital televisions, and other digital display devices are being called upon to display images with increased graphics requirements. For example, in computer systems, software applications typically include graphical user interfaces (GUIs) which place increased burdens on the graphics capabilities of the computer system. Further, the increased prevalence of multimedia applications also demands computer systems with more powerful graphics capabilities. Modem digital television systems, including interactive television systems, also have increased video display requirements.
The anticipated merging of the digital television and computer system markets will require new technology to efficiently integrate digital television, computer systems, and Internet/communications technology. One problem in particular that will need to be addressed is how to efficiently combine multiple video and graphics sources when each source has its own independent frame rate. Therefore, a graphics system and a method capable of providing increased performance while efficiently combining multiple video and graphics sources with independent frame rates are desired.
The present invention comprises an integrated memory/graphics controller (also referred to herein as an Integrated Memory Controller, Interactive Media Controller, or IMC) which utilizes a novel video display refresh list (VDRL) system and method for presenting data on a display device or video monitor, such as a computer video monitor or television. The memory/graphics controller of the present invention minimizes data movement and manipulation for video display and thus greatly increases system performance.
In one embodiment, the VDRL comprises a plurality of pointers that point to span line segments of graphics data stored in memory (e.g., a frame buffer or system memory). The VDRL is xe2x80x9cexecutedxe2x80x9d by reading the graphics data pointed to by the pointers and then outputting the data to a display device. Advantageously, by using pointers in the VDRL, the graphics data need not be stored in contiguous memory locations (as with a traditional frame buffer).
When an application program indicates that a particular object is to be displayed (e.g., a rendered three-dimensional graphics object), the graphics controller (e.g., under the control of the device driver) may then create an object display list subroutine (ODL) that is responsible for rendering the object into memory at the correct frame rate. Multiple ODLs may exist any one time. For example one ODL may be responsible for drawing the three-dimensional object previously mentioned into memory at ten frames per second, while a second ODL may be responsible for receiving and performing MPEG decompression on a digital video stream from the Internet at fifteen frames per second. In one embodiment, after an ODL completes drawing a frame of the object into memory, the ODL may be configured to set an interrupt or may change a value in an input/output register or memory location indicating that a new frame of the object is ready to be displayed.
Since ODLs may be allocated for more than one buffer, advanced graphics features such as motion estimation and multiple-frame animation may be implemented using the ODLs. In one embodiment, ODLs with multiple buffers may have static pointers that each point to one of the buffers. The ODLs may then have a dynamic pointer which indicated which static pointer is to be used. Thus, animation be accomplished by drawing each frame to be animated into its own buffer, and then rotating through the buffers using the dynamic pointer.
The graphics controller may be further configured to utilize draw display lists (DDLs) to incorporate the objects drawn by the ODLs into the image draw when the VDRL is executed. The DDLs may be repeatedly executed at whatever speed the graphics controller is capable of. For example, the ODLs may be executed at their predetermined frame rate (i.e., ten and fifteen times per second in the example above), the VDRL may be executed at the display device""s refresh rate (e.g., 70 times per second), and the DDL may be executed as often as possible given the hardware of the graphics controller. The DDL may comprise a plurality of pointers to buffers in memory. When the DDL is executed, it is configured to draw into one of the buffers. When drawing into the buffers, the DDL may execute graphics commands and may incorporate graphics data from the ODL memory buffers also. For example, a particular DDL may draw a window into memory, wherein the window comprises a digital video ODL (DV-ODL), and three-dimensional rendered ODL (3D-ODL) overlaid on top of the digital video, and a number of two-dimensional controls. The DDL may comprise pointers to the two ODLs, instructions on how to overlay them, and instructions for drawing the controls.
When a DDL has completed an execution cycle, like the ODLs it too may be configured to assert an interrupt bit. The interrupt bit may be used by the graphics controller when constructing the VDRL to ensure that the scan line segment pointers in the VDRL point to the most recently completed buffer drawn by the DDL.
In one embodiment, the graphics controller of the present invention comprises an IMC, which includes advanced memory, graphics, and audio processing capabilities and performs pointer-based display list video operations according to the present invention. The UMC includes numerous significant advances which provide greatly increased performance over prior art systems. The IMC of the present invention preferably interfaces to a CPU bus and one or more high speed system peripheral buses, such as the PCI bus, USB (Universal Ser. Bus), or IEEE-1394 (Firewire) bus. The IMC includes one or more symmetric memory ports for connecting to system memory. The IMC also includes video outputs, preferably RGB (red, green, blue) and horizontal and vertical synchronization signal outputs, to directly drive the display device. The IMC also may include a separate video port for other video I/O. The IMC also preferably includes an audio/telephony subsystem for digital audio and telephony communication.
The IMC transfers data between the system bus and system memory and also transfers data between the system memory and the video display output. Therefore, the IMC architecture of the present invention eliminates the need for a separate graphics subsystem. The IMC also improves overall system performance and response using main system memory for graphical information and storage. The IMC system level architecture reduces data bandwidth requirements in multiple ways for graphical display since the host CPU is not required to move data between main memory and the graphics subsystem as in conventional computers, but rather the graphical data resides in the same subsystem as the main memory. In addition, the IMC may contain a novel parallel compression and decompression engine for the reduction of ODL, DDL, and VDRL commands and data including associated texture and image maps. Therefore, for graphical output, the host CPU or DMA master is not limited by the available bus bandwidth, thus improving overall system throughput.
The IMC of the preferred embodiment includes a bus interface unit which couples through FIFO buffers to an Execution Engine. The Execution Engine may include a digital signal processor (DSP) core which performs compression and decompression operations, as well as texture mapping, and which also assembles display refresh lists according to the present invention. The Execution Engine in turn couples to a Graphics Engine which couples through FIFO buffers to one or more symmetrical memory control units. The Graphics Engine is similar in function to graphics processors in conventional computer systems and includes line and triangle rendering operations as well as span line interpolators. An instruction storage/decode block is coupled to the bus interface logic which stores instructions for the Graphics Engine and the Execution Engine.
A Video Display Refresh List (VDRL) Engine is coupled to the Graphics Engine and the one or more memory control units. The Display Refresh List Engine in turn couples to a display storage buffer and then to a display memory shifter. The display memory shifter couples to separate digital to analog converters (DACs) which provide the RGB signals and the synchronization signal outputs to the video monitor; The Video Display Refresh List Engine includes a novel display list-based method of transferring video data or pixel data from the memory to the video monitor during screen refresh according to the present invention, thereby improving system performance. In one embodiment, the VDRL Engine, referred to in this embodiment as a Window Assembler, both assembles and executes the display refresh list.
An anti-aliasing method is applied to the video data as the data is transferred from system memory to the display screen. An overlay method may also be applied to the video data for rendering overlaid objects. The internal graphics pipeline of the IMC is optimized for high end 2D and 3D graphical display operations, as well as audio operations, and all data is subject to operation within the execution engine and/or the graphics engine as the data travels through the data path of the IMC.
Video screen changes or screen updates are preferably performed using the following operations. First, in response to software executing on the CPU, such as applications software or interactive television applications, the video driver executing on the CPU generates a video driver instruction list which includes-screen update and/or graphics information for displaying video data on the screen. The video driver instruction list may be provided to the Execution Engine in the graphics controller or IMC. The Execution Engine examines the video driver instruction list and generates a list of graphics and/or memory commands to the Graphics Engine. Thus the Execution Engine constructs a complete list of graphics or memory operations to be performed in response to desired screen change information.
If the Execution Engine receives an Assemble Display Refresh List command from the video driver, the Execution Engine assembles a display refresh list comprising a plurality of pointers which reference video data in the system memory that is to be refreshed to the video monitor. The plurality of pointers reference memory areas in the system memory which store video or pixel data for respective objects that appear on the display screen. The pointers reference portions of the data on a scan line basis, and the pointers are used to read out the data on a scan line basis during screen refresh. The use of a display refresh list for screen refresh operations greatly reduces data traffic as compared to prior art computer architectures and thus provides significantly improved performance.
The VDRL Engine of the present invention uses the display refresh list constructed by the Execution Engine to perform pointer-based or display list-based video refresh operations according to the present invention. The display refresh list operations enable screen refresh data to be assembled on a per window or per object basis, thereby greatly increasing the performance of the graphical display. The VDRL Engine includes memory mapped I/O registers storing values which point to various buffers or object information memory areas in system memory comprising video or graphics display information. The IMC includes an ID pointer register which points to a Windows ID list. The Windows ID list comprises a list of pointers for each of the windows or objects appearing on the display screen. Each respective pointer in the Windows ID list points to respective windows workspace memory areas corresponding to the window. The windows workspace areas specify data types, color depths, 3D depth values, alpha blending information, screen position, window attributes, etc. for the respective window or object on the screen. Each windows workspace area also includes static and dynamic pointers which point to the location in system memory where the pixel data for the respective window or object is stored. Each windows workspace area also optionally includes a pointer to a color composition matrix for color indexing on a per object or per window basis, a secondary workspace pointer for rendering overlaid objects, and optional slope information for rendering non-rectangular objects.
The Execution Engine utilizes the information in the Window Workspace buffer, as well as information received from the software driver regarding screen changes, to assemble a display refresh list in system memory. When a screen change occurs, such as a new window displayed on the screen, the Display Refresh List Engine uses the display refresh list to determine where in the linear or xy memory space the data resides as well as how many bits per pixel the window requires, how to map the color space, and the necessary xy rectangle extents and window priority. This information is used during the screen refresh to display the various windows or objects on the screen very quickly and efficiently. Thus, the video display can be updated with new video data without requiring any system bus data transfers, or the movement of data from offscreen to onscreen memory locations, which may be required in prior art computer system architectures.
The Execution Engine dynamically adjusts the display refresh list or assembles a new display refresh list for movement of objects and changes in relative depth priority which appear on the display. Thus when an object or window is moved to a new position in the screen, or is popped or pushed relative to another window, the data comprising the object is not transferred to another location in memory, but rather only the display pointer address is changed in an object information area or in a new display refresh list. This provides the effect of moving data from a source to a destination, i.e., a bit blit (bit block transfer), without ever moving the object in memory. This provides a tremendous performance increase over conventional bit blit operations commonly used in graphical subsystems. This also greatly reduces memory bandwidth requirements in a unified memory environment.
The video data stored in system memory is preferably stored in a plurality of memory areas, which may or may not be contiguous. The plurality of display memory areas each preferably store video data corresponding to video objects or windows, at least a subset of which are displayed on the video monitor. Thus the present invention is not required to maintain, and preferably does not maintain, a single frame buffer which contains all of the video data for display on the video screen. Rather the video data for the various windows and objects is stored in respective memory areas in the system memory, and pointers assembled in the display refresh list are used to reference this data during screen updates. Thus, data is not required to be moved in or out of a frame buffer to reflect screen changes, but rather in many instances either the video data for a respective window or object is changed, or only the pointers in the display refresh list are manipulated, to affect a screen change.
The IMC of the present invention uses a virtual color depth technique which optimizes the use of system memory, i.e., uses only the amount of system memory required for each application and each window on the display. Low end applications may only require a minimum amount of memory, such as one bit per pixel, whereas high end applications may require more memory per pixel. In the IMC architecture, memory is used on a xe2x80x9cper application basisxe2x80x9d where only the memory that is actually required is used for each application or window. For example, a simple text application may only use a single bit per pixel while a complex 3D application may require as much as 128 bits per pixel. According to the present invention, both applications reside on the same display simultaneously and each uses only the memory required for its respective window size and pixel depth. This virtual color depth method also reduces the memory bandwidth requirements in a unified memory environment.
The present invention further includes a novel method for presentation of a specific object or window overlaid on top of another object or window without destruction or off-screen copy requirements used by prior art methods of object overlay. This overlay method animates objects with transparency in a very efficient manner. The method used multiple windows workspace areas for the objects and also includes multiple pointers in the display refresh list to retrieve data for the two objects (foreground and background) involved in the overlay. The method then performs a color comparison as the video traverses through the IMC during screen refresh.
The pointer-based display list method of the present invention also allows screen refresh rate edge anti-aliasing and filtering method to be applied to video data on the fly as data is being refreshed on the screen. As discussed above, data is read from the system memory according to the current display refresh list, which is continually updated as screen changes occur. As the data traverses the serial FIFO shift registers in the IMC, the edge anti-aliasing process occurs. The edge anti-aliasing method is enabled by attribute bits located in the window workplace flags in system memory for each window. Each window workspace includes an enable flag which indicates whether smoothing is desired. If the flag is set, filtering is performed using the edge anti-aliasing filter method. The screen refresh display input data which is being provided to the display FIFO buffers is compared against predetermined threshold values which represent the delta change in intensity or color. If the method detects a large change, the method performs automatic blending and smoothing of the area around the abrupt change in intensity. This anti-aliasing method can be used in conjunction with the secondary window overlay method. Thus the edges of overlaid objects can be smoothed during the video refresh operation.
The pointer-based display list video refresh system and method of the present invention removes system bottle-necks and greatly increases performance. The system and method of the present invention uses a high level graphical protocol between the CPU and the IMC which reduces bus traffic and greatly increases bus bandwidth. Thus many changes to video data in the system memory are accomplished by pointer manipulation, not by the transfer of video data across the system bus. This is a significant advance over the operation of current video/graphics architectures and their associated graphic display subsystems.
Therefore, the present invention comprises an integrated memory and graphics controller including a novel pointer-based display list refresh system and method which provides greatly increased performance over prior art designs.